Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Jetpack AI: Adding translation support using Chrome's Gemini AI mini #41724

Merged
merged 8 commits into from
Feb 14, 2025
Merged

Jetpack AI: Adding translation support using Chrome's Gemini AI mini #41724

merged 8 commits into from
Feb 14, 2025

Conversation

mwatson
Copy link
Contributor

@mwatson mwatson commented Feb 11, 2025 •
edited
Loading

Fixes #41212

Proposed changes:

  • Adds support for Chrome AI (Gemini mini) for translations
  • This is behind a feature flag and in beta (see test instructions for more)
  • This should be fairly seamless for users, meaning if a translation cannot be done by Gemini it will fall back to our original translation method
  • See test instructions for more info / limitations

Other information:

  • Have you written new tests for your changes, if applicable?
  • Have you checked the E2E test CI results, and verified that your changes do not break them?
  • Have you tested your changes on WordPress.com, if applicable (if so, you'll see a generated comment below with a script to run)?

Jetpack product discussion

No product discussion but the PT is here: pe4Cmx-30R-p2

Does this pull request change what data or activity we track or use?

No

Testing instructions:

  • Two options:

  • Once you have a token you can continue with the next steps:

  • Check out this branch, add the following snippet to enable beta features, the feature flag, and add your token in place of [ your trial token goes here ]:

define( 'JETPACK_BLOCKS_VARIATION', 'beta' );
add_filter( 'ai_chrome_ai_enabled', '__return_true' );

$token_transient_name = 'jetpack-ai-chrome-ai-token';
if ( ! get_transient( $token_transient_name ) ) {
	set_transient(
		$token_transient_name,
		"[ your trial token goes here ]",
		3600
	);
}
  • This snippet should also make it so you don't have to modify your sandbox to return your trial token. However for the duration of the trial we will have to ask WPCOM for the token. Currently there is no general purpose token, which is why this setup is required for testing.
  • Once this is set up you should be able to translate using Chrome.
  • Open the javascript console and enable debug mode.
  • Create a new post and add some text. Try to translate it as you normally would into a supported language
  • As long as the language is supported by Chrome (all of my testing was done by translating to Spanish), you should not see the translation streamed from OpenAI as you normally would in the console. However the text should be replaced with the language you selected. There may be a small delay while Chrome downloads the language pack.
  • Revert /reject the changes and observe that the paragraph goes back to the previous language
  • Try to translate it to an unsupported language (French or German are what I normally use) and observe that the translation goes through OpenAI and you see the translation streamed in the console.
  • Try out various languages and translations, revert, change back, etc, and take not of any weird or unexpected behavior.
  • Disable the feature flag and observe that translating to a supported language now goes through OpenAI with a streamed response

@mwatson
Jetpack AI: Adding translation support using Chrome's Gemini AI mini
b4b91f1 
* Adds optional support for translations using Chrome's built in AI
* Will fall back to our standard OpenAI translations if Gemini can't be found or can't translate
* Puts the entire feature behind a beta feature flag
* API key currently read from WPCOM, changes TBD there
@mwatson
changelog
6aac5e0
@mwatson mwatson requested a review from a team February 11, 2025 16:54
@mwatson mwatson self-assigned this Feb 11, 2025
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 11, 2025 •
edited
Loading

Are you an Automattician? Please test your changes on all WordPress.com environments to help mitigate accidental explosions.

  • To test on WoA, go to the Plugins menu on a WordPress.com Simple site. Click on the "Upload" button and follow the upgrade flow to be able to upload, install, and activate the Jetpack Beta plugin. Once the plugin is active, go to Jetpack > Jetpack Beta, select your plugin, and enable the change/add-chrome-ai-translate-support branch.

  • To test on Simple, run the following command on your sandbox:

    bin/jetpack-downloader test jetpack change/add-chrome-ai-translate-support

Interested in more tips and information?

  • In your local development environment, use the jetpack rsync command to sync your changes to a WoA dev blog.
  • Read more about our development workflow here: PCYsg-eg0-p2
  • Figure out when your changes will be shipped to customers here: PCYsg-eg5-p2

@github-actions github-actions bot added [Block] AI Assistant [JS Package] AI Client [Plugin] Jetpack Issues about the Jetpack plugin. https://wordpress.org/plugins/jetpack/ RNA labels Feb 11, 2025
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 11, 2025 •
edited
Loading

Thank you for your PR!

When contributing to Jetpack, we have a few suggestions that can help us test and review your patch:

  • ✅ Include a description of your PR changes.
  • ✅ Add a "[Status]" label (In Progress, Needs Team Review, ...).
  • ✅ Add a "[Type]" label (Bug, Enhancement, Janitorial, Task).
  • ✅ Add testing instructions.
  • ✅ Specify whether this PR includes any changes to data or privacy.
  • ✅ Add changelog entries to affected projects

This comment will be updated as you work on your PR and make changes. If you think that some of those checks are not needed for your PR, please explain why you think so. Thanks for cooperation 🤖

The e2e test report can be found here. Please note that it can take a few minutes after the e2e tests checks are complete for the report to be available.

Follow this PR Review Process:

  1. Ensure all required checks appearing at the bottom of this PR are passing.
  2. Choose a review path based on your changes:
    • A. Team Review: add the "[Status] Needs Team Review" label
      • For most changes, including minor cross-team impacts.
      • Example: Updating a team-specific component or a small change to a shared library.
    • B. Crew Review: add the "[Status] Needs Review" label
      • For significant changes to core functionality.
      • Example: Major updates to a shared library or complex features.
    • C. Both: Start with Team, then request Crew
      • For complex changes or when you need extra confidence.
      • Example: Refactor affecting multiple systems.
  3. Get at least one approval before merging.

Still unsure? Reach out in #jetpack-developers for guidance!

Jetpack plugin:

The Jetpack plugin has different release cadences depending on the platform:

  • WordPress.com Simple releases happen semi-continuously (PCYsg-Jjm-p2).
  • WoA releases happen weekly.
  • Releases to self-hosted sites happen monthly. The next release is scheduled for none scheduled (scheduled code freeze on undefined).

If you have any questions about the release process, please ask in the #jetpack-releases channel on Slack.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 11, 2025 •
edited
Loading

Code Coverage Summary

Coverage changed in 2 files.

File Coverage Δ% Δ Uncovered
projects/plugins/jetpack/extensions/blocks/ai-assistant/ai-assistant.php 0/95 (0.00%) 0.00% 38 💔
projects/js-packages/ai-client/src/hooks/use-ai-suggestions/index.ts 0/84 (0.00%) 0.00% 3 ❤️‍🩹

3 files are newly checked for coverage.

File Coverage
projects/js-packages/ai-client/src/chrome-ai/factory.ts 0/39 (0.00%) 💔
projects/js-packages/ai-client/src/chrome-ai/suggestions.ts 0/30 (0.00%) 💔
projects/js-packages/ai-client/src/chrome-ai/index.ts 0/0 (—%) 🤷

Full summary · PHP report · JS report

Coverage check overridden by I don't care about code coverage for this PR Use this label to ignore the check for insufficient code coveage. .

@dhasilva dhasilva force-pushed the change/add-chrome-ai-translate-support branch from 538e5b5 to 3f6d6a6 Compare February 11, 2025 22:28
@dhasilva dhasilva force-pushed the change/add-chrome-ai-translate-support branch from 3f6d6a6 to a5dfba6 Compare February 11, 2025 22:38
dhasilva and others added 3 commits February 12, 2025 12:37
@dhasilva
move declaration to regular types file
77c9d61
@dhasilva
Merge remote-tracking branch 'origin/trunk' into change/add-chrome-ai…
7b03068 
…-translate-support
@mwatson
Better handling of PrompType payloads and convert markdown to HTML (a…
6ac9102 
…nd then back) so translations work better. This fixes the issue where a full AI generated article could not be translated to a different language.
@mwatson mwatson requested a review from vianasw February 12, 2025 21:09
@mwatson mwatson added the I don't care about code coverage for this PR Use this label to ignore the check for insufficient code coveage. label Feb 13, 2025
@mwatson
Copy link
Contributor Author

mwatson commented Feb 13, 2025

I added the "don't care" tag since we discussed adding tests and decided against it (for now).

dhasilva
dhasilva approved these changes Feb 13, 2025
View reviewed changes
Copy link
Contributor

@dhasilva dhasilva left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works well, nice!

vianasw
vianasw approved these changes Feb 14, 2025
View reviewed changes
Copy link
Contributor

@vianasw vianasw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good. Thanks for addressing the issue with the other flow 👍

@mwatson
Copy link
Contributor Author

mwatson commented Feb 14, 2025

Looking good. Thanks for addressing the issue with the other flow 👍

thank you for finding it! 🥹

@mwatson mwatson merged commit ffa32e9 into trunk Feb 14, 2025
69 checks passed
@mwatson mwatson deleted the change/add-chrome-ai-translate-support branch February 14, 2025 16:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vianasw vianasw vianasw approved these changes

@dhasilva dhasilva dhasilva approved these changes

Assignees

@mwatson mwatson

Labels
[Block] AI Assistant I don't care about code coverage for this PR Use this label to ignore the check for insufficient code coveage. [JS Package] AI Client [Plugin] Jetpack Issues about the Jetpack plugin. https://wordpress.org/plugins/jetpack/ RNA [Type] Task
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Implement translation with Built-in AI API
3 participants
@mwatson @vianasw @dhasilva